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The unique character and needs of the Comprehensive 
nt Monitoring (CAM) system has led to the creation of new 
programs and new techniques. Thus, earlier manual strategies 
re subject to error) for the handling of the large banks of 
1 objectives and test items, have been supplanted by a 
se computer program that stores the objectives and 
, prints out selected objectives and their associated 
for inspection and, finally, constructs and prints the 
ests in ready- to-use form. The attitudes of students and 
toward CAM are discussed in detail and a more complete 
this subject is being prepared. The arrangement of test 
ronologica lly (in terms of teaching order) versus random, 
appear to be a significant variable. The report concludes 
ief review of steps being taken to secure demonstrations of 
variety of situations, by involving individual 
state agencies across the country. (GS) 
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The long range goal of the Project for Comprehensive 

Achievement Monitoring Is to develop a new technique for the mea- 

surement of student performance in the classroom. To reach this 
goal extensive invention, research, demonstration, and dissemina* 
tlon have been necessary. The Third Annual Report will discuss 
the accomplishments of the Project in each of these areas. The 
major emphasis will be concerned with discussing the current level 

of acceptability of students and teachers of the CAM system. The 

success of the Project ultimately rests on the support from these 
two groups. The appendices offer a collection of detailed infor- 
mation about the publications, p tiejpants and research instru- 
ments of the Project;. 




- 2 - 



T nvention 

Several of the tasks of the Project for Comprehensive 
Achievement Monitoring ore unique to its organisation. No other 
achievement testing program has comparable ones. To facilitate the 
accomplishment of these tasks the Project has had to invent new 
computer programs and new techniques. This section will briefly 
describe two inventions of the Project. 

Objective and Question Bank Computer Progr am 

One of the major features of the CAM i9 the need to 
have available a large number of behavioral objectivec to describe 
the goals of a course and even a larger number of test questions 
to measure student performance of the objectives. Several manual 
strategies have been developed for the selection and duplication of 
the objectives and questions. Each of these strategies has improved 
these operations but none of them is without the serious limitation 
of constant errors occurring at each stage in the processes. 

To overcome the problems in these operations the P?o- 
ject has developed a computer program which will perform the clerical 
functions of these operations while allowing the teacher or experi- 
menter the flexibility to make the final choite of objectives and 
to assign them to the monitors. 

The computer progta* operates in three phases. The 
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f:"rat phase will allow the Project to keypunch the questions and 
their associated objectives on computer cards . The computer cards 
can then be stored on a magnetic computer tape. The tape storage 
allows a large number of questions to ba stored economically and 
prevents unwanted rearrangement of the text of the questions. The 
second phase of the computer program allows the teacher or researcher 
to request a preliminary listing of all of the possible objectives! 
and their associated questions, which might be relevant to a course. 
The list is printed out in a format which is easy to read and can be 
thoroughly studied. Hie third and final phase of the program allows 
the questions to be assigned to specific positions on specific ntoni- 
tor forms and prints the forms out in a format which can be used 
directly as tests along with the computer card required by the com- 
puter program used for analyses. 

The computer program is the current solution to the 
problem of quickly and accurately storing, selecting and printing 
the CAM monitors. The necessary flexibility has been built into 
the program to allow its use in many courses. The accuracy of the 
computer ensures that the clerical tasks which are critical to the 
operation of CAM are performed without errors. 

item-free Ability Estimation 

CAM monitoring is a highly efficient and comprehensive 
methodology for measuring the achievement of groups of students on 




- 4 - 



specific performance objectives. Hie usefulness of the CAM In 
measuring individual student growth and achievement is not as highly 
developed. The number of questions which any one student is Asked 
and can answer at each test administration is usually very small. 

The information about each student is therefore limited. 

The Project is cooperating with several faculty mem- 
bers at the University of Massachusetts in the near future in an 
effort to apply a new technique called Item-free ability estimation 
to gain more information about individual students. The originator 
of the statistics for the technique was G. Rasch but a similar, 
more general model has been developed by Birnbaum. The technique 
may allow CAM to estimate individual grow h during the year. 
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Research 

Comprehensive Achievement Monitoring relies very 
heavily on statistical techniques and recently developed tools in 
the field of educational measurement. CAM uses these ideas in a 
combination which has not been tried before. The result is po- 
tentially a powerful strategy for the evaluation of student a- 
chievement. However, the use of such new ideas necessitates in- 
vestigating their characteristics and discovering the influence 
of different variables on the quality of the measures of achieve- 
ment. The following section describes the results of several of 
the surveys and experiments performed during the la9t year. 

The Attitude of Students Toward CAM 

The student is an integral part of the CAM system. 

The repeated measurement of the students’ achievement will con- 
tinue to be valid and reliable only as long as the students make 
an honest attempt to answer the cues t ions. To maintain student 
interest in the monitoring procedure and to utilise the informa- 
tion collected about an Individual student, teachers are en- 
couraged by the Project to involve students In the interpretation 
of the results and in the use of individual reports to direct their 
studying. Each student was asked, by the twe-page questionnaire 
Included in Appendix C, for his opinion about CAM. The mean of 
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the responses, for each course, was calculated. The results vere 
encouraging In several ways, although there are areas Identified 
by the students In which improvement Is needed. 

The mean student reaction to the general question, 
"l like to take CAM tests (13)," for the courses hnveved around 
neutral. Although this does not register enthusiasm for CAM it 
does not indicate a general, strong dislike for a program which 
is primarily concerned with testing. Tho students identified 
ab having performed in the upper third of the class were more 
positive than their classmates. The composition of the CAM tests 
used in the 1968-1969 school year covered the entire breadth of 
the course. A student with a lower ability may not have appre- 
ciated the importance of his improvement in the course and this 
developed a negative opinion of CAM as well as tha course. In 
answer to the question "There aren’t many things in this subject 
that are interesting (8)," the students reacted in a neutral way 
also. 

Much of the usefulness of the CAM system to the 
9tudent is directed away from the testing situation towards the 
interpretation of the individual computer reports. The students 
on the average were neutral with "A lot can be learned by Just 
taking the CAM tests (3)" and slightly disagreed with "The CAM 
test 9 don't help me to know whet 1 should StuJy next (4)." Both 
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questions refer to the testing situation directly which may contain 
vestiges of the anxiety of a typical testing situation. It is not 
clear to students that they learn or are being prompted to recall or 
to try problems measuring aspects of their .ourse. The positive 
aspect cf the CAM procedures for the student is located in the use 
of the computer output. When asked l,r fhe computer report of my CAM 
tests helps me figure out what material I haven’t learned very 
well (7)» n the average response was more than slightly agree. The 
students are positive toward the use of the output and find it 
helpful. 

The operation of the Project in the schools is good. 

The students are very positive in their reactions to "There is 
usually enough time to answer all of the CAM test questions (1)," 
and negative to "There are too many CAM tests given in this course 
(2)." The arrangements made in allocating time for testing and the 
frequency of testing (usually biweekly) are acceptable to students. 

It is important to students that the mechanics of the CAM system 
are smooth. One aspect which the students do not perceive as ade- 
quate is the turn-around time on the report of the CAM analysis as 
indicated by thei. neutral, on the average, to "I get my CAM test 
results back too late to do any good (1-4)." Either a local data 
processing system or a data communication system would greatly reduce 
the delay in returning the analysis to students. 
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One very positive opinion held by students is their 
confidence in the importance of the results to the teacher in 
planning lessons for the class. They reacted quite positively to 
"The CAM. tests are good because the teacher can find out what the 
class needs help on (20)," One of the strengths of the CAM system 
is its ability to provide the teacher with information useful to 
planning the course* For the students to appreciate the value of 
the information to the teacher is a major accomplishment for CAM. 

A more complete report of these results is being prepared. 

The Attitude of Teachers Toward CAM 

Many different activities aie necessary to the oper- 
ation of CAM in the school. The classroom teacher is often the 
person who is responsible for the leadership in this operation* If 
someone else in the school has been made responsible for the man- 
agement of the mechanics of the monitoring system the teacher is 
still the key figure in communicating the results of the CAM pro- 
cedure to the students and must himself make use of the information 
in adjusting his teaching to the circumstances in student learning 
suggested by the CAM analysis. 

An interview or a questionnaire was administered to 
each of the teachers involved with the Project last school year. 

The teachers were asked questions about their experience with the 




-9- 



preparation of materials for the CAH system, i.e. , test questions 
and objectives. They were asked to describe the procedures they 
used in preparing the CAM monitors, scheduling students to the 
tests, administering the monitors to the students, and scoring and 
analyzing the CAM data. Further, and most important, is their re- 
action to the procedures. What they felt were the advantages and 
disadvantages of the periodic monitoring and what they felt were 
the students 1 opinions. The information gathered from the teachers 
is summarized in this section. It is intended to provide a picture 
of the current acceptability of the CAM system in the public schools 
as seen by the teachers. 

The preparation of behavioral objectives is by far 
the most time consuming component of the CAM process. Teachers 
have generally spent a h to 8 week portion of the summer on this 
task. The results of the first year of objective writing has often 
been revised during the second year of operation. The revisions 
have often been in the domain of refining materials and deleting 
extraneous objectives and have not taken more than two weeks. 

Rarely, in the experience of the Project, has the set of objectives 
developed for a cours * not covered more than the amount of work 
possible by 99% of the students. 

To facilitate the initiation of more schools into the 
CAM system the Project had attempted to collect at least an initial 
set cf objectives for each of the major subject matter areas to the 
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elementary and secondary school levels. Many of the participants 
in the Project in the future will be expected to choose the as- 
sistants of the preliminary set of materials for monitoring. The 
Project has already experimented with the strategy of providing a 
set of objectives to teachers as a first approximation to the con- 
tent of their course. The computer program for storing objectives 
will serve as a tool to facilitate the distribution of these 
materials . 

The preparation of test questions for the CAM pro- 
cedure Iras typically also been done during the summer. Exceptions 
to this rule are available, i.e., where a teacher systematically 
added both objectives and test questions to a cumulative file for 
the year and had to merely revise or organize them when the summer 
came. About 40 to 200 hours were used in the writing and editing 
of the test questions for a course. Often the teachers stated that 
they did not get repeated use of the questions which they would 
normally write for their usual testing program although they would 
spend an equivalent amount of time. The CAM system has not resulted 
in an increase in the amount of time. It requires that the mate- 
rials be used many times which distributes the value of the teacher's 
efforts over more of the year and even allows the teacher to sys- 
tematically use the materials from one year to the next. Although 
the writing must be performed at the same time so that all of the 




- 11 - 



questions are available for the monitors, benefits may be derived 
from this strategy because the teacher has a clearer picture of the 
structure and continuity of the course* 

The Project has made the test question writing situa- 
tion even more of reasonable step for the teachers in the schools by 
developing test question bank to measure student performance ob- 
jectives, This procedure has proved useful in the schools. It is 
possible to begin CAM monitoring with a much smaller commitment of 
time from the teacher beyond the usual classroom duties. Several 
schools have already begun CAM monitoring by developing tests from 
the test questions in the test question bank. 

The administration of the CAM monitors to the students 
is one of the purely mechanical operations of the system but in 
several ways is the most controversial. Teachers are asked to care- 
fully hand a different test to each student in the class. Each 
test must be accompanied by an answer sheet which is accurately 
marked with the student’s number, the CAM monitor form number, and 
the student’s responses to the questions. The work is clerical am! 
strategies are constantly being sought to minimize teacher time 
needed for this job, The most immediate solution has been to enlist 
the help of a teaching aid. The aid will typically spend 2 hours 
per test administration sorting and marking the answer sheets for 
about one hundred students. With this task taken care of, teachers 
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usually feel that the CAM system is easier to carry out than 
the usual classroom testing because of its regularity and the use 
of computers to score answer sheets. Also the use of many test 
forms at the same time eliminates the problem of cheating. Another 
positive characteristic of the administration is that it can be 
done during a class period. This makes the stystem comparable in 
the type of time allocation and much easier to arrange than the 
longer standardized tests. 

When the teachers were asked to report about the 
likes and dislikes of their students towards CAM they described 
a broad range of characteristics and the students 1 reaction to 
them. The regularity of the CAM system is seen as a positive 
aspect - of the Project by students because they know what to expect. 
The content of the monitors is known in general and their format 
is familiar to them after a few test administrations. Ihe teachers 
describe the effect on the students by saying that the systematic 
organization seems to ease the tension which an unfamiliar exam or 
a final exam would create. 

The teachers also report that the students are in- 
fluenced in what they study by the CAM procedures in two ways. The 
first is to point out to the student the areas of the course where 
he must review before the end of the course. The areas are very 
specific for the CAM feedback directs the student to performance 
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objectives. The second is more unusual for a testing program. 

The students in some cases are prompted to study ahead in the 
course topics which have not yot been covered. The student has a 
repeated opportunity in the testing situation to test his skills 
and knowledge on new problems. The course therefore takes on a new 
dimension of interrelatedness because recently acquired competences 
can be used in related problem situations, 

Teachers have very specific characteristics which 
they find advantageous about the CAM system. First, the information 
provided by the computer analysis of the longitudinal data serve 
the teacher as an important tool in evaluating the effectiveness 
of their own teaching. By pointing to performance objectives which 
the students have not learned adequately even after teaching the 
teacher is able to review these topics during the current year and 
modify his teaching and assignments during subsequent years. Second, 
CAM has a more unique feature of allowing the teacher to see the 
retention of the performance objectives by the student and to modify 
the course to refresh student comprehension in areas easily for- 
gotten, The notion of retention should play an important part in 
the teacher's planning the structure of a course. 

The combination of the first two advantages and the 
ability of the CAM system to pretest students' performance create 
the third advantage of the system for teachers. It allows the 
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teacher to sys tematically develop the curriculum for a course. 

Many of the teachers in the Project have significantly modified 
an old curriculum or made major second year changes in a new curric- 
ulum. Usual classroom testing can not match the CAM technique in 
this domain and teachers have used it in many cases. 

The general change in most students 1 attitudes towards 
testing would be a fourth advantage for CAM. The easing of the 
pressure often caused by the usual testing situation is considered 
a positive contribution to the teaching-learning situation. The 
CAM monitoring allows the students repeated opportunities to 
demonstrate his performance. The CAM system also requires the 
students to remember material throughout the course which stimulates 
the student to really learn material because he will be expected 
to use it over and over again on the CAM monitors. 

If the administration and the correction of the CAM 
monitors is carried out smoothly, a fifth advantage of the CAM 
system becomes the effective use of data processing equipment to 
support the teacher’s work. The potential types of analysis open 
when the data are collected systematically and are available to 
the computer can greatly improve the effectiveness of teachers. 

The disadvantages to the CAM system mentioned by the 
teachers include teaching to the test, some restriction of the teachers 
usual flexibility, too many tests, the lack of questions requiring 
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any thing except recall, and not enough time to use the CAM Informa- 
tion. Each of these criticisms Is being considered in the future 
research and development for the Project. Interestingly enough, 
several of the criticisms of CAM are connected with the organization 
of the current school programs. Not enough time to use the analysis 
provided by CAM is a function of the allocation of time in the 
school. It reflects the lack of conviction in the worth of eval- 
uation in the modern curriculum. For CAM to function at its most 
profitable level the use of many of the innovative ideas of the 
field of education will have to be brought into focus in the ar- 
rangement of priorities in the schools. 

Variables in the Monitoring Procedure 

The variables of monitor question arrangement, the 
prediction of classroom achievement, and the meaningfulness of 
total monitor test score were considered in experiments carried 
out during the year. The purpose of the experiments was to in- 
vestigate the sensitivity of the monitoring procedure to certain 
variables which are usually thought to influence the quality of 
the information provided by CAM. 

The affect of monitor question arrangement . The 
questions on CAM monitors might be arranged on a form in several 
different ways depending upon the criterion used in the assignment 
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of a question to a position on the form. The technique often used 
in test construction to assign items to positions is that of random 
assignment, i.e., randomly so that no systematic bias is introduced 
into the structure of the test. This technique has almost ex- 
clusively been used in the design of the CAM monitors so that sub- 
sequent analysis of results can be performed under the assumption 
of random assignment. 

Teachers have expressed a concern for the possible 
discouragement which students may feel because of the lack of 
questions which they can successfully answer on any CAM monitor, 

They felt that by arranging the questions in the order in which the 
students were taught the material the question tested, i.e*, chro- 
nologically that the students would have the satisfaction of finding 
questions which they could more often answer at the beginning of 
the monitor. They would therefore be less discouraged, A second 
more technical reason for arranging the questions in the chrono- 
logical fashion is the possibility for the teacher to score the 
results himself and have a clear cut-off point after which the 
student would not be expected to know the answer to questions and 
before which he should have answered all of the questions. The 
answer sheet when corrected becomes a very readable and inter- 
pretable record of the students 1 performance. The student might 
even use the answer sheet as his summary of his achievement in the 
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The experiments to investigate the comparison between 
the achievement results for students taking the monitors with the 
randomly arranged questions versus those with the chronologically 
arranged questions are described in detail in TM-21 and TM-22. 

Tiie conclusion reached in the context of two courses in high school 
mathematics is that the arrangement is not a significant variable 
in the context of the CAM monitoring procedure. This result en- 
courages the schools to proceed with the use of tests in either a 
random or a chronological of questions. Informally gathered im- 
pressions of teachers suggest that the students usually can iden- 
tify questions which they should be able to answer because they have 
covered the appropriate material in their class already. They 
merely go through the questions on the monitors and try at least 
all of the questions which they are able to answer because of their 
studying in class. 

Aptitude test scores as criteria for stratification . 
Nine reference tests of cognitive ability, as discussed in TM-18, 
were administered during the beginning of the school year. Their 
usefulness in stratifying students into different monitoring groups 
was analyzed at the end of the year. The analysis consisted of a 
correlation of the nine ability scores with the posttest score on 
a test composed of a number of CAM test items measuring the entire 
content of the course. The correlations for the course are die- 
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played in Table TM-22.2 and are all below 0.360. The correlation 
of this relatively low magnitude probably does not justify the 
expense of administering the aptitude tests singly. A step-wise 
regression was used to predict the final score from the nine ability 
test scores. Eight of the nine scores (the ninth not being sig- 
nificant) yielded a multiple correlation coefficient of 0.430. As 
a set, the nine cognitive ability tests chosen were only moderate 
predictors of posttest achievement and under normal circumstances 
do not justify the cost of administration and scoring for purposes 
of scheduling students » 

Usefuln e ss cf tota l test score . It was initially 
hoped that the total monitor test score from period to period would 
provide statistically useful information about the progress of 
individual students. The analyses performed on the tests used last 
year in CAM monitoring indicate that total test scores of individual 
students should not be used in their present form without extreme 
caution. They are useful to teachers in a cumulative way to red- 
flag students who just are not progressing in their achievement in 
the course as v/ell as to announce the presence of students who have 
essentially mastered all of the expected material. 
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Demonstration 

One of the most potent techniques for persuading In- 
dividuals or organizations to try an innovative practice is to show 
them an operational demonstration in a natural setting. This stra- 
tegy has been used in many contexts besides education and is a popu- 
lar idea in education. The Project for Comprehensive Achievement 
Monitoring has been actively seeking out opportunities to sat-up 
demonstrations of the CAM technique in schools across the country. 
The search has taken two major forms. First, the Project has re- 
quested the participation of individual teachers or schools in the 
CAM procedure. Second, the Project has nade specific proposals to 
various agencies at the state le'el to support a demonstration of 
the CAM monitoring technique in the evaluation of other educational 
projects or in the systematic design of a group of demonstration 
schools . 

Demonstration Schools 

Through a variety of ways the involvement of schools 
in the Project has steadily inci eased over the last three years. 

The schools participating in the project during the first year were 
entirely research oriented. The schools which have joined the Pro- 
ject of the succeeding years have accepted the CAM technique as a 
substantive tool in education which must be trial nested in the 
specific environment of their school. The increase in terns of 
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numbers of students, teachers, schools, end states involved in the 
CAM monitoring technique is very encouraging. The specific statis- 
tics for the different numbers involved are displayed in Table 
AR-3.1. 

Pro posals for the Denonstrat ion of CAM 

A major emphasis in the Project has been the prepara- 
tion of proposals to several state agencies for extensive demonstra- 
tions of the capabilities of CAM in educational program evaluation 
and in the public schools. The New York State Department of Educa- 
tion, Division of Research and Evaluation, requested that a proposal 
be written to evaluate programs funded by the Urban Education section 
of the Department, The proposal has been submitted (TM-20) and dis- 
cussions of specific work is in progress. A second proposal (TM-23) 
has been developed for the Commonwealth of Massachusetts. The goal 
of the proposal would be the establishment of a system of demonstra- 
tion schools in the state which would serve to promote the use of CAM. 
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Table AR-3.1 Growth of Involvement In CAM 

Measured by Several Criteria 
for Three Years 



Criterion 




Number involved 


1 9 7 


19 C 8 


1969 


States 


A 


6 


7 


Schools 


5 


8 


17 


Teachers 


16 


23 


75 


Students 


800 


1100 


2600 
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Dissemination 

Dissemination has taken a broad spectrum of approaches 

to communicate to the educational community an awareness of CAM 
and to encourage its use. 

Tec hnical Memor anda 

The Project has made a systematic effort to rake its 
findings available to the people interested in the technical as- 
nects of achievement. The series has grown steadily and is listed 
in Appendix D. 

Professional Meetings and Speeches 

Major presentations of the notions of CAM have been 
made at our professional meetings, three of which were national 
and the other state. The purpose of the participation of the Pro- 
ject in these meetings was to inform particular organisations of 
the potential of CAM monitoring to meet sore of their goals. The 
organisations consisted of educational researchers, educational 
measurement specialists, and teachers of mathematics. The recep- 
tion of the groups to the ideas of CAM was generally very suppor- 
tive. 

The sta :e level meeting was the Annual Meeting of 
the Educational Research Association of Xew York State. The Pro- 
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ject cooperated with members of the State Department of Education 
in preparing an extensive manual describing in detail the operation 
of the Project. This guide is included in the appendix I). It 
was used as the basic reading for the day-long oresession, whose 
objective was to train participarls in CAM techniques. 

In addition to the meetings described above, various 
members of the Project staff have hao the opportunity to speak to 
various individuals and groups during the past year. The groups 
have included teachers' organizations in more than half a dozen 
states. Other occasions have brought the staff in front of audi- 
ences of state education department personnel of several states 
including New York. 

Journal Articles 

During the year six articles have appeared in pub- 
lished form. The purpose of four of these articles has been to 
describe the potential of the computer programs which have been 
developed by the Project. Another article was directed at the ex- 
perimental results concerning concepts used in CAM. The last 
article was an overall description of the general goals and opera- 
tion of the Project, During the coming year several wore articles 
are planned. 




